Predictive model generation

ABSTRACT

Models are generated using a variety of tools and features of a model generation platform. For example, in connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is provided through a graphical user interface a structured sequence of model generation activities to be followed, the sequence including dimension reduction, model generation, model process validation, and model re-generation. Historical multi-dimensional data is received representing multiple variables transformed to be maximally predictive for at least one outcome variable to be used as an input to a predictive model of a commercial system, model development process is validated for at one or more sets of such variables and enabling a user of a model generation tool to combine at least two of the variables from the sets of variables.

BACKGROUND

This description relates to predictive modeling.

Predictive modeling, for example applied to targeted marketing, refersto modeling (a) which of a company's customers would likely buy a new oradditional product (that is, would be susceptible to a cross-sell orup-sell effort); or (b) which prospects from a population of potentialcustomers would be likely to accept an offer for a product or service(called acquisition by response or look-alike); or (c) which existingcustomers are most likely to cancel a current service (called retentionor chum reduction); or (d) trigger points for behavior outside thenormal range; or (e) to estimate the expected value or magnitude of apredicted outcome. Modeling is typically done by an analyst who iseither an employee of a company or of an external consulting or servicebureau. The analyst uses his experience and skill to create a custommodel using available model building software applied to currentlyavailable data. The cost and accuracy of the model depend on the abilityof the analyst, the time permitted to develop the model, the quality ofthe data used, and the performance of the model development softwaretools.

When the model is to be used to select prospects, who will be thetargets of, say, a direct mail advertising campaign, the time andexpense of creating the model and the quality of the selection producedby the model are important considerations.

SUMMARY

In general, in one aspect, a machine-based method includes, inconnection with a project in which a user generates a predictive modelbased on historical data about a system being modeled: selectingvariables having at least a predetermined level of significance from apool of potential predictor variables associated with the data, to forma population of predictor variables, extending the population to includenon-linear interactions of variables, extending the population toinclude linear and non-linear extensions with remaining previouslyexcluded variables, generating a possible model of the extendedpopulation of variables using a subsample of the data, determiningwhether the model generalizes to the data other than the subsample, ifso, applying the possible model to all of the data to generate a finalmodel, and cross-validating the final model using random portions of thedata.

Implementations may include one or more of the following features:Information displayed to the user of the size of the pool of predictorvariables. The user interface enables the user to point and click toreduce or extend the size of the pool of predictor variables retainingonly the most significant predictor variables. The user interfaceenables the user to invoke an automatic machine-based method to selectthe class of models most suitable to the pool of predictor variables forthe associated dataset. The user interface enables the user to point andclick to adjust the model selection criterion to retain only the mostpotent variables for the target goal. The user interface enables theuser to point and click to cause display of information about the modelperformance. The information about the model performance includes atleast one of: a statistical report card with a link to the statisticalreport chart, a lift chart with a link to the lift chart, a responsecomparison chart for each decile for each predictor variables in themodel, a link to the response comparison chart, measures of NPV orprofitability impact, and a link to NPV/profitability comparison charts.A course of action is determined that will yield the most positive NPVoutcome. The determination includes selection of a mix of channel andproduct combinations. The determination includes predicting retention incombination with response rate to predict NPV. The invocation of thelink to the statistical report card causes display of the statistics ofmodel performance. The invocation of the link to the lift chart causesdisplay of a non-cumulative lift chart. The invocation of the link tothe response comparison chart causes display of a response chart foreach predictor variable in the model for each segment of interest. Auser is enabled to choose interactively at least one performancecriterion change or transformation or interaction of variables toimprove the fit of the model. Means of determining whether the modelgeneralizes to the data other than the subsample, if so, applying thepossible model to all of the data to generate a final model, andcross-validating the final model using random portions of the data. Theuser interface enables the user to select at least one validationdataset and invoke a model process validation method. The user interfaceenables the user to point and click to cause display of informationabout the model process validation. The information about the modelprocess validation includes at least one of: a statistical report cardwith a link to the statistical report chart, a cumulative lift chartwith a link to the cumulative lift chart, a non-cumulative lift chartwith a link to the non-cumulative lift chart. The invocation of the linkto the statistical report card causes display of the statistics of modelprocess validation. The invocation of the link to the cumulative liftchart causes display of a cumulative lift chart. The invocation of thelink to the cumulative lift chart causes display of a non-cumulativelift chart. A user is enabled to choose interactively at least oneperformance criterion change or transformation or interaction ofvariables to improve model validation process. The user interfaceenables the user to select at least one machine automated modeldevelopment process applied to the entire dataset for a validated modelprocess. The user interface enables the user to point and click to causedisplay of information about the performance of the validated modelprocess applied to the entire set of historical data. The informationabout the model performance for two independent data subsets includes atleast one of: a statistical report card with a link to the statisticalreport chart, a cumulative lift chart with a link to the cumulative liftchart, a non-cumulative lift chart with a link to the non-cumulativelift chart. The invocation of the link to the statistical report cardcauses display of the statistics of model process validation. Theinvocation of the link to the cumulative lift chart causes display of acumulative lift chart. The invocation of the link to the cumulative liftchart causes display of a non-cumulative lift chart. The final model andthe model process validation results are stored persistently.

In general, in another aspect, a machine-based method includes, inconnection with a project in which a user generates a predictive modelbased on historical data about a system being modeled, for example,displaying to a user a lift chart, monotonicity, and concordance scoresassociated with each step in a step-wise model fitting process.

Implementations may include one or more of the following features. Theuser is enabled to observe changes in the fit of the model as variablesassociated with the data are added or removed from a predictor set ofthe variables. The user is enabled to terminate the fitting of the modelwhen the fitting process reaches an optimal point.

In general, in another aspect, a machine-based method includes, for apredictive model based on historical data about a system being modeled,generating measures of the fit of the model to samples of the data, thefit measures being generated separately in percentile segments.

Implementations may include one or more of the following features.Variables associated with the data are identified for which the modeldoes not produce at least a predetermined degree of fit for at least oneof the percentiles. A user is enabled to choose interactively at leastone transformation or interaction of variables to improve the fit of themodel

In general, in another aspect, a machine-based method includes receivingfrom separate sources, sets of variables representing historical dataabout a system being modeled, and enabling a user of a model generationtool to combine at least two of the variables from the sets ofvariables.

Implementations may include one or more of the following features.Enabling the user to combine the variables includes providing a userinterface that enables the user to identify the variables to becombined. The system includes behavior of prospective or currentcustomers with respect to two or more products or services of a companyand the method also includes adjusting variables to normalize responserates across products or services with different response rates, usingthe computed propensities as indices of the scores.

Other aspects include media that bear instructions to perform themethods, apparatus to perform the methods, and other methods associatedwith those described above.

Advantages

Among the advantages of these features and aspects are one or more ofthe following. Project management of the workflow of model development,including scored list generation, report production, and modelmaintenance is achieved. Complete documentation and project replicationor project refinements are easily performed at a later date, even byanalysts without prior involvement. Control is maintained in a highvolume production environment without requiring analyst initiative. Theefficiency of orchestrating the workflow is improved so as to minimizedevelopment time. The system documents both successful and unsuccessfulapproaches as the analyst applies serial testing of alternative datatreatments and alternative techniques, enabling easy replication. Theunifying graphical user interface assures that the analyst uses themodel generation platform's integrated techniques correctly. Theinterface controls the staging of the successive techniques and reducesinefficiencies. The analyst is discouraged from making inappropriatedecisions at choice points by enabling only choices that fit the datatypes. The datasets of interest for such models involve numerous recordswith a large number of attributes, many of which are redundant orirrelevant, so their elimination results in improved analysis, reducedcomplexity and improved accuracy. For typical datasets with a largenumber of attributes, the systems transforms variables, whose raw formlack predictive power, or are misleadingly complex, into designervariables producing efficient models with strong predictive power. Thesystem provides effective management, interpretation, and transformationof large numbers of complex attributes. The system readily constructspowerful models from disparate datasets with parallel or sequentialstages that facilitate the achievement of targeting marketing goals withmultiple products or offers, and cross product. The system assuresconsistently successful models that are optimally developed andthoroughly tested even when executed by analysts without years ofexperience. Reliable and informative measures of model robustness areprovided. A detailed perspective on the distinguishing characteristicsof customer segments can be used to design creative marketing media.

Other aspects, features, and advantages will be apparent from thedescription and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a model development platform.

FIG. 2 is a schematic diagram of a computer system.

FIG. 3 is an exemplary design layout of a model project database

FIG. 4 is a schematic diagram of steps in a model development

FIG. 5 is a schematic workflow diagram of a model development process

FIG. 6 is a schematic workflow diagram of a variable transformationprocess

FIG. 7 is a schematic workflow diagram of a missing data imputationprocess

FIG. 8 is a schematic workflow diagram of a Bayesian renormalizationprocess

FIG. 9 is a schematic workflow diagram of a harmonic concatenationprocess

FIG. 10 is a schematic workflow diagram of a dimension reduction process

FIG. 11 is a schematic workflow diagram of a model generation process

FIG. 12 is a schematic workflow diagram of a model process validation

FIG. 13 is a schematic workflow diagram of a model ensemble process

FIG. 14 is a schematic workflow diagram of a customer insight analysis

FIG. 15 is a schematic component diagram of modules in a modeldevelopment process

FIG. 16 is a schematic diagram of workflow in a dataset preparation anda model development

FIG. 17 is a screen display of a user interface.

FIGS. 18 a and 18B are an exemplary representation of a model projectentry form.

FIG. 19 is an exemplary graphical user representation of a predictorvariable display and a transformation editor.

FIGS. 20A and 20B is an exemplary graphical user representation of aninvocation of an interaction tree to analyze predictor variables.

FIGS. 21A, 21B, 21C, 21D, and 21E show exemplary graphical userrepresentations of a variable response function, a variabledistribution, a partition tree of a variable, and the application of thevariable editor to transform variables with an cross-product interactionand showing an addition of the constructed interaction variable to alist of predictor variables.

FIG. 22 is an exemplary graphical user representation of a dimensionreduction interactive dialog form showing five (5) stages of variableexclusion.

FIGS. 23A, 23B, and 23C show an exemplary graphical user representationof a model selection interactive dialog showing a selection option formodel variable persistence; an exemplary graphical user representationof a gains chart and some statistical results; and an exemplarygraphical user representation of a persistence chart by deciles for atypical continuous variable showing an approximate equivalence ofaveraged values for target and non-targets.

FIG. 24 is an exemplary graphical user representation of a modelselection interactive dialog showing decision options available aftermodel parameter optimization.

FIGS. 25 a, 25B, and 25C are an exemplary graphical user representationof a gains chart and some statistical results comparison of a predictivemodel developed from a sample dataset and an application of the samemodel to a validation dataset.

FIG. 26 shows an exemplary graphical user interface to a summary reportof a model project.

FIGS. 27A, 27B, 27C, and 27D are an exemplary graphical userrepresentation of a model project insight interactive dialog showinghyperlinks to a target profile and corresponding chart of key factors;and an exemplary graphical user representation of a model projectinsight chart showing target and non-target contributions for two keyfactors.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 illustrates a sequence 10 of development and deploymentactivities that enable a user to generate a predictive model. Thedevelopment of the model is organized on a project basis. A successfulmodel typically produces a series of updates and modifications as themarket or commercial system to which the model is directed evolves andchanges. Organizing the activities on a project basis reduces wastedtime and improves the quality of the models. By enforcing a carefullymanaged project paradigm, models can be generated, updated, changed,reviewed, and deployed in a high-volume production process, at lowercost, and with better results.

The model development and deployment activities may begin with adatabase 12 that covers historical events associated with the systembeing modeled. For example, if the system being modeled is the behaviorof customers of a vendor, the database 12 may include records thatidentify each customer, demographic information about each customer,information about products that the customer owns, and communicationsthat the customer and the vendor may have had. Project planning 14 maybe the initial step in model development and can be a step that isrevisited from later activities. After the initial project planning,data in the database 12 may be set up 16, and modeled 18. Then the modeldevelopment process (which may be expressed in software) is validated20. Then the model (which may be expressed in software) that isgenerated using the validated model development process may be evaluated22. After evaluation, further project planning may be necessary anddeployment of the model may be scheduled 24. Deployment could include,for example, applying the model to live data to generate predictiveinformation (for example, a list of people who the model predicts willbe more likely to accept a new product offering or to predict the mostprofitable combination of channel and target selection) or deliveringthe model to a client for use with current and future data that issimilar to the originally analyzed data referenced in the database. Allof the activities of FIG. 1, and others, are managed and enabled by amodel development platform.

As shown in FIG. 2, the model development platform may be implemented insoftware 30 running on a workstation 32 that includes a microprocessor34, a random access memory 36 that stores instructions and data that areused by the microprocessor to run the program, a mass storage system 38that holds the historical data of the system being modeled 40, metadata42 related to the historical data, generated by the model generationplatform and used in generating the model, project data 43, model data41, an operating system 45, and the model development platform software44 related to the management of the development activities as projects,among other things. Input/output devices 46 enable a user to interactwith the model generation platform to generate a model by a series ofsteps to be described later.

Although a single workstation is shown in FIG. 2, the system could beimplemented on multiple workstations on a network, such as a local areanetwork or a wide area network or implemented in client-server modeusing a server on a network. Access to the features of the modelgeneration platform software could be permitted using an applicationservice provider (ASP) model of distribution such as a web service overthe internet. Or the model generation platform software could bedistributed by downloading or on portable storage media.

The use of the model generation platform software is based on a projectparadigm. Whenever a user wishes to develop, modify, update, copy, orenhance a model, he must do so within the context of a project. To keeptrack of and control the use of projects, the model development platformmaintains the project data in the form of a table the definition 50 ofwhich is shown in FIG. 3. The values in the table maintain the integrityof the model development process. Among the fields for each projectrecord are an ID 52, a name 54, a project goal 56, a pathname 58 to thehistorical data (called “ProjectDataset” in FIG. 3), a pathname 60 tometadata and model data (called “ProjectDataDictionary”), and a targetvariable 62 for the model selected to be the dependent or predicted oroutcome variable of the model. Other entries in the table will beunderstood with reference to the discussion below. Thus, the modeldevelopment platform automatically stores structured project informationthat captures a state of the project at successive steps in generatingthe model.

Report Generation

Another feature of the model development platform is to generate reportsfor tracking by model analysts and for executive decisions. The reportsmay be in the form of web browser displayable hypertext markup language(HTML) documents (text and graphics) and extended markup language (XML)representations that can be stored digitally and re-filled andre-purposed. Such reports facilitate going back to prior projects andprior models. The reports also show the choices made when buildingmodels.

The sequence of activities involved in generating a model is shown inFIG. 4. The model development platform enforces the performance of theseactivities in the order shown: dataset exploration 70, dimensionreduction 72, candidate model generation 74, model process validation76, final model selection 78, list scoring 80, and customer insightgeneration 84. Variable transformation 82 is an aspect of datasetexploration, dimension reduction, and model generation. At each step,statistical tables, graphs, and charts are generated in XML or directlyinto web-browser viewable form. Hotlinks to the HTML documents aremaintained in the project table.

Types of Reports

In the dataset exploration stage, aspects of the historical data may beexamined in terms of the predictor variables. By predictor variable, wemean a potential covariate or independent variable of a model. In amanner explained later, the analyst may select analysis tests using acheckbox or point and click interface. The software then automaticallycomputes, for the variable: (a) variable response graphs in the variableeditor showing which values or trends are most closely associated withthe dependent variable in order to select, redefine, or interactvariables to predict more efficiently and more accurately the targetoutcome, (b) development sample distributions illustrating for eachpredictor variable the target response frequency for each target value,for example, the value “1” for a buyer and the value “0” for a non-buyerwith descriptive statistics of central tendency and dispersion, (c)interaction trees showing successive binary recursive partitioning of asingle variable or of the set of predictor variables in descending rankorder of the most significant log likelihood (G²) variables that bestclassify the prospects (within the data) from the non-prospects. Suchanalysis informs the appropriate variable transformation to maximizepredictive power to supplement the automatic transformation of variablesto that purpose preceding or following dimension reduction.

In the stages of model generation, model process validation and finalmodel selection, the user by means of point and click or checkbox caninvoke software that automatically computes (d) candidate modelstatistical features tables which display ranked significant variablesand goodness-of-fit as measured by the c-statistic (i.e., the area underthe receiver-operator-characteristic (ROC) curve), (e) non-cumulativegains charts showing the response rate as percentile per decile comparedwith random selection as a base rate (100%), (f) cross-sample comparisonstatistics for the candidate model that compare the development sampleand validation datasets to determine the robustness and generality ofthe candidate model, (g) cumulative and non-cumulative gains charts thatare similarly computed for the development sample and validationdatasets to reveal whether the candidate model is sufficient in terms ofperformance and robustness beyond the sample dataset on which it wasbased, (h) comparison statistics and charts for the key variables andmodel scores in the development dataset and scoring dataset computed toassess the validity of applying the model to score the campaign list,

In the stage of target profile generation, the user by means of pointand click or checkbox can invoke software that automatically computes(i) profiles and charts of key variables ranked in order of significanceare computed to provide insight into the target population of buyers. Atthe end of a cycle of model development, the user by means of point andclick or checkbox can invoke software that automatically computes (j) asummary report including all the above as hotlinks to the correspondingHTML documents along with the initially collected information at themodel processing entry point.

Graphical User Interface

To make model development efficient, model projects are developed alonga managed sequence of steps interactively with the analyst using areentrant graphical user interface. The user can invoke the modelingprocess repeatedly, for example, to review outcome response functionsfor individual predictor variables, or recursive partition trees forindividual variables or for the set of predictor variables; to modifythe predictor variable pool, to create and filter additional variables,or to modify the candidate model. The integrity of the model ismaintained by the project database and the guidance supplied to theuser.

To begin a project, the project requirements, constraints, and goals areentered as a first step. An historical dataset is selected with itsaccompanying metadata. The metadata may include but is not limited tovariable names, variable descriptions, variable definitions which mayinvolve transformation of other variables, variable measurement typesuch as ordinal, categorical, or interval, and variable model statussuch as predictor variable or excluded variable. This entry step isfollowed by sample/validation set allocation, sample data exploration70, model dimension reduction 72, model generation 74, model processvalidation 76, final model generation 78, which may include modelensembles, and model parameter reporting or list scoring 80 (see FIG.4).

The user's choices are guided by the model generation platform. Forexample, if too few targets are included in the sample dataset forreliable results, a warning is issued so corrective action can be taken.Included in the initial dataset exploration is variable interactiondetection. Such interaction effects reflect the complex interrelationsof the types of data available and may have powerful influences on thechoice of the optimal model. The model process is re-entrant so that theuser can make revisions without restarting. However, the user interfaceprovides a goal-oriented framework for achieving successful models. Eventhough the user may re-enter any activity of model generation at anytime, once having reentered, his work is guided by the software tofollow the predefined sequence of project steps from the point ofreentry.

Thus, there is an effective scheme of user interaction that permits bothuser decisions to reenter the process at an earlier point andsoftware-guided sequences of steps to be followed when reentry occurs.The sequencing of these steps has an important impact on modelperformance. For example, modeling before dimension reduction mayproduce inefficiency (the computationally intensive modeling process maytake time to complete), loss of robustness (variables may be includedthat spuriously inflate the performance of the model and degrade thevalidation), and loss of predictive power (raw untransformed variableswill mislead the modeling process by diluting predictive power).

As shown in FIG. 5, the model generation platform controls the modelingprocess as a workflow through a sequence of stages, primed byinformational tables, graphs and charts, with the goals of obtaininginsight into the target population and then generating an actionablescoring list of prospects that belong to the population. The businesslogic of the modeling process workflow is embedded into the state chartof FIG. 5. Details of the workflow will be discussed later. The blocksrepresent activities in the model generation process. A new modelproject is initially set up 90 with at least one outcome variableselected, then a subset of the full historical data as a samplepartition 92. Subsequently the record attributes are reviewed 94,displayed 96 and edited 98 and/or automatically transformed intopredictive variables 100. The set of variables is reduced to those thatare maximally predictive by dimension reduction 102. Then a model methodis selected 104 and fitted to convergence 106 with the sample dataset.The model generation process is validated 108. With the validated modelgenerated process a final model is selected 110 and persisted along withother model generation reports 112. The persisted models are used singlyor in combination 114 to maximize expected return on investment thenapplied to score lists 118 after first validating the applicability ofthe model ensemble to the list 116. The arrows represent the mainsequence that the software permits the analyst to follow.

Complex Data Preparation Transforms

Even when aggregated and summarized, the resulting variables may notreveal their full predictive power. As shown in FIG. 6, the modelgeneration platform provides automated and interactive features totransform variables by imputing missing values 120, compensating forsparsely sampled values 122, smoothing variables with grouping andsplines 124, and straightening variables with mathematical transforms126, such as squares, logs, and hyperbolic tangents. Information aboutthe type of measurement a variable represents is retained in themetadata. The model generation platform metadata includes informationabout the nominal, ordinal, or interval scalability of variables,enabling differential and hence more accurate treatment of data. Themodel generation platform tracks predictor variables and theiruntransformed antecedents, and provides an editing facility for applyingtransforms and visualizing the distributions of the variables relativeto the target populations.

The model generation platform persistently stores the metadata for thevariables imported in the development dataset using XML transforms

The variables from the originating development dataset are typed forstrength of measurement (e.g., nominal, ordinal, interval/continuous)with descriptive labels for ease of interpretation. For active variablesa status is assigned (P, XP, T, XT) to reflect whether the variable isin the pool of predictor primary variables (P) or transformed variables(T), or has been excluded from either (XP or XT) with the appliedtransformation in the variable definition field. There are severaloptions for transformations to apply for nonlinear continuous variablesand non-uniform categorical variables:

-   -   Logarithmic transform: If((:X>0), Log(:X),0)    -   Square: : X {circumflex over ( )}2    -   Square root: If(: X>0, Root(: X, 2), 0)    -   Box-Cox power (r) transform: If(:X>0, Exp(:r*Log(:X)), 0)    -   Nonlinear interaction: : X * :Y    -   Partition: If(LowerBound <=: X|: X<UpperBound, 1, 0)    -   Missing value: If(Is Missing(: X), 0, If(: X ==1, 1, 0))

Certain transformations may be inappropriate or may be without effectfor predictor variables of a particular measurement type. For example,squaring a binary categorical variable {0,1} would have no effect andtaking the log would be inappropriate. The model generation platformautomatically disables the inappropriate options when the analyst isediting variables. In the editing process during the model developmentprocess, variables can be examined, redefined, and shifted in or out ofa pool of predictor variables using a model project variable editor.

Missing Data Imputation

In the recursive interaction analysis, missing values are imputed by arandom method using the distribution of the variable in question. Wherethe percentage of missing values of a particular attribute is small,typically 10% across the sample dataset, then single imputation by mean,by hot-deck substitution, or by regression can be employed, although thedistribution statistics are biased. For continuous variables withmissing values, a method of regression imputation can be used toestimate predictor variable missing values from the regression on theknown predictor variables. However, the method of multiple imputationcan be applied by combining results from single imputations withoutresulting bias to the distribution statistics for sparse data providedthe missing data are either missing at random or missing completely atrandom.

With reference to FIG. 7, the first step 130 is to determine if thevalues that are missing, for example, from a categorical variable arenon-ignorable, i.e., whether an absence of response is equivalent to ananswer of ‘No’. If so, then the missing value can be replaced 132without loss of information by ‘No’ or its numerical equivalent. Next, adetermination 134 needs to be made whether the values missing from therecord are missing at random in a statistical sense. If not, thenimputing a missing value may introduce bias so the missing value is notmodified 136. Records without missing data may be segregated to avoidloss since for most modeling methods records with missing data for asingle predictor variable are ignored. If, on the other hand, the datais missing at random then specific techniques can be applied dependingon the type of variable as determined from the metadata of the datadictionary 138. For nominal or ordinal variables a categorical missingdata imputation 140 can be made. For continuous variables an intervalmissing data imputation 142 can be made.

Bayesian Renormalization

Conventional modeling approaches can be significantly misled by spuriousassociations between the available variables and the target variable. Todetermine the deep structure in the typical sample datasets used inpredictive modeling, it is important to avoid attractive artifacts thatsuggest strong predictive power. The model generation platform achievesthis by a form of Bayesian renormalization, which is appliedautomatically to significant categorical variables, prior to binning,but also can be selected for manual application to identifiedcategorical variables in the variables palette.

The model generation platform's predictor variables palette displays thevariables present in the development dataset according to variablestatus, i.e., whether the variable is considered currently as apredictor variable, or has been excluded from present consideration as apredictor variable. When a displayed variable has been selected, theassociated metadata, e.g., description and definition, is shown alongwith a hyperlink to a graph of the variable.

In the algorithm described by FIG. 8 for the case of a multi-valuednominal variable, the model generation platform calculates the responsefrequency of the dependent (predicted) variable and ranks those variablevalues by response frequency. The calculation of response frequencybased on the development dataset is adjusted iteratively by Bayesiananalysis based on the variable's a priori response frequency to estimatethe underlying probability. Initially, the mean response frequency iscomputed, or, for variables with a functional dependence onsubvariables, a regression surface is calculated 150. An advancedversion of this approach is to regress to a hyperplane fitted to X usingother covariates. To reduce the variability of an unstable estimator ofresponse frequency for a particular dataset, X_(m), denote the vector ofresponse frequency prediction (at the observed sites or at newlocations) by F(X_(m)).

The response frequency for a given value is regressed toward the meanresponse frequency based on the number of observations or toward aregression hyperplane determined by subvariables underlying the variableof interest 152. Denote the reweighted estimate of F(X_(m)) where theweights, ω(m), give rise to a weighted sample as {X_(m), ω(m)}. Aconvergence test is then performed on the recomputed transcategory meanor regression plane 154. If the iteration has not converged within areasonable approximation the process is repeated; otherwise, theadjusted response frequencies are used in subsequent operations. Thehigh response frequency of a sparsely populated value bin will bereduced and the low response frequency of a sparsely populated value binwill be increased. However, the response frequency of a heavilypopulated value bin will remain essentially the same.

Harmonic Concatentation Transformation

Categorical values associated with similar target variable responsefrequencies (subject to sampling considerations) are conjoined into asmaller set of categories, which then form a statistically optimalclassification mapping. The implementation may include prior imputationof missing data for the variable, or prior Bayesian renormalization,then sequential binary partitions with respect to the target orindependent variable using the likelihood-ratio statistic, G², which fora candidate input attribute, A_(i), and the target attribute, T, isgiven by:G ²(A _(i) ; T/z)=2·(ln 2)·E*(z)·MI(A _(i) ; T/z)  [7]

-   -   where E*(z) is the number of records associate with node z and        MI is the measure of mutual information. So that G² is actually        twice the natural log entropy or change in the entropy,        sigma-log(p), for each observation, where p is the probability        attributed to the response that occurred.

The procedure for obtaining the optimum number of groupings is describedin FIG. 9. The likelihood-ratios (G² values) are computed for the binarysplits for the predictor attribute at the given node for the targetoutcome attribute 160. Then the binary split that maximized the value ofG² is chosen 162. If one or more of the resulting nodes cover sufficientproportion of the number of records in the sample as determined byk-fold cross validation then the process iterates 164 until completion.Sufficient proportion should be in excess of 5% of the sample size andfor variables for which this procedure generates improved models,maximization to yield not more than ten (10) intervals in the range ofthe variable containing not less than ten (10) percent of the totalresponse. Then the records for the predictor attribute each node areassigned the value of the likelihood ratio, G² 166.

Complex Dimension Reduction

A model can be viewed as a hyperplane in multidimensional space definedby the set of variables that best separates (within the space) thetargets (for example, the customers of a company who would be mostlikely to respond favorably to an offer for a new product) from thenon-targets (those customers who would not respond favorably). Modeldevelopment includes a process of removing non-significant variables(dimensions) of the space until the hyperplane is defined by a coherentset of significant variables (predictor variables) and theirinteractions (in the real world data, the initial variables may not beorthogonal). Simple dimension reduction (variable elimination) caninvolve removal of variables having limited predictive power. Butvariables that alone have only limited predictive power may, throughinteraction, have a significant predictive ability. Complex dimensionreduction involves filtering to retain such variables and constructingcomplex, pluripotent variables to move the modeling process over thefitness landscape to a global optimum.

To do this the model generation platform uses a cascade of filters(F_(n)) to screen out non-significant predictor variables:F ₀(G(x ¹ ₁ , . . . , x ¹ _(n)))−>H(x ² ₁ , . . . , x ² _(m))  [1]where G(x¹ ₁, . . . , x¹ _(n))=1 and H(x² ₁, . . . , x² _(m))=1 are thehyperplanes defining the respective successive models.

The flowchart in FIG. 10 provides an example of such cascade offiltering operations on the set of predictor variables. The first filter170 reduces the dimensionality of the modeling space by eliminatingvariables, x¹ _(k), for which the density, D(x¹ _(n)), is less than somefixed constant, C₁. (These are variables which have not or can not beprocessed for missing data imputation.)F ₁(H(x ² ₁ , . . . , x ² _(m)))−>J(x ³ ₁ , . . . , x ³ _(p))  [2]

-   -   where p is less than or equal to m, H(x² ₁, . . . , x² _(m))=1        and J(x³ ₁, . . . , x³ _(p))=1 are the hyperplanes defining the        respective successive models. The dimensionality of the dataset        has been reduced from m to p.

In the second filtering stage 172, a subspace is iteratively generatedby including only significant variables, e.g., x² _(k), whoseprobability of non-contribution, [1−Pr(y|x² _(k))], is less than a fixedconstant, C₂.F ₂(H(x ² ₁ , . . . , x ² _(p)))−>J(x ³ ₁ , . . . , x ³ _(q))  [3]

-   -   where H(x² ₁, . . . , x² _(p))=1 and J(x³ ₁, . . . , x³ _(q))=1        are the hyperplanes defining the respective successive models.        In effect, the second filter reduces the dimensionality of the        modeling space from p to q.

In the third stage 174, the subspace, X_(Q), is expanded by includingall significant cross-products, x^(j) _(k)*x^(p) _(q), where x^(j) _(k)and x^(p) _(q) are in X_(Q), then applying a filter 176 to retain onlysignificant variables, e.g., x^(l) _(k), whose probability ofnon-contribution, [1−Pr(y|x⁴ _(k))], is less than a fixed constant, C₃.F ₃(J(x ⁴ ₁ , . . . , x ⁴ _(q)))−>K(x ⁵ ₁ , . . . , x ⁵ _(r))  [4]

-   -   where J(x⁴ ₁, . . . , x⁴ _(q))=1 and K(x⁵ ₁, . . . , x⁵ _(r))=1        are the hyperplanes defining the respective successive models.        In effect, the third filter 176 reduces the dimensionality of        the modeling space from q to r.

In the fourth stage 178, the augmented subspace, X^(#) _(Q) is furtheriteratively expanded with all the subspaces, x^(j) _(k)*x^(p) _(q)+z^(r)_(s), which are now significant where z^(r) _(s) are from X_(M)−X_(P)then applying a filter 180 to retain only significant variables.

In the fifth stage 182, the augmented subspace, X^(##) _(s), is furtheraugmented with all the cross-products, x^(j) _(k)*z^(r) _(s), wherex^(j) _(k) are from , X^(##) _(S), and z^(r) _(s) of from X_(N)−X_(M)then applying a filter 184 to retain only significant variables, e.g.,x⁵ _(k), whose probability of non-contribution, [1−Pr(y|x⁵ _(k))], isless than a fixed constant, C₄.F ₅(K(x ⁵ ₁ , . . . , x ⁵ _(r)))−>L(x ⁶ ₁ , . . . , x ⁶ _(s))  [5]

-   -   where K(x⁵ ₁, . . . , x⁵ _(r))=1 and L(x⁶ ₁, . . . , x⁶ _(r))=1        are the hyperplanes defining the respective successive models.        In effect, the fifth filter 184 reduces the dimensionality of        the modeling space from r to s.

The filters adjust the dimensionality of the modeling space by mappinginto transformed variables with truncated ranges or mapping intocompound (nonlinear) variables (x⁴ _(q)=x³ _(i)*x³ ₁) whose probabilityof non-contribution, [1−Pr(y|x¹ _(k))], is less than a fixed constant,C_(j). The resulting hyperplane represents an efficient classificationmapping. The set of predictor variables is then adjusted to include thevariables passing through the sequence of filters resulting in anupdated collection of predictor variables 186 available for constructingpredictive models.

Modeling Choice

In the context of a model development project, the mathematical orstatistical approach should be matched to the type of variable and todistribution variability. The model generation platform for the modelingstage described in FIG. 11 has access to a variety of models through alibrary 192 of model functional types, for example, but not limited tologistic, general location, and decision trees.

The model generation platform generates predictive analytic model that,for example, retrospectively classify prospects and non-prospects,starting with the selection of the appropriate model type 190. Forexample, when the dependent variable has nominal measurement type andthe required assumptions about the dataset distributions are minimal,logistic models are desirable provided extreme outliers can be avoided.The model generation platform filters out such extreme outliers.Following dimension reduction the number of significant variablestypically has been reduced into the range where a stepwise nominallogistic proceed can be efficiently applied. More specifically, givenlogit[Pr(Y ₁=1|X ₁ , . . . , x _(k))]=C(0)+Sum(j=1,k)C(j)X _(j)  [6]where X_(j) represent original attributes, transformed nonlinearinteraction variables, or transformed by a segmentation of a predictorvariable, then a maximum-likelihood method can be used to estimate thecoefficients, C(n). In a product-optimization project, for example, astepwise linear discriminant analysis (general location model) may bepreferred, if the more stringent assumptions about the responsedistributions of the variables can be justified.

Consider the general case of the multinomial distribution,Pr[Y_(i)|X_(j), Z_(k)], where Y_(i) is a vector of i dependentvariables, X_(j) is a vector of j continuous variables and Z_(k) is avector of k categorical variables. But if there is a single dependentbinary variable, Y, which has a Bernoulli distribution, i.e.,Pr[Y=1]=e ^(L)/(1+e ^(L))  [7]

-   -   where L is linear in X_(j) and Z_(k); however, X_(j) is assumed        to be normally distributed. If so, then a two-group discriminant        analysis can be applied. If not, then the standard logistic        regression, which is not sensitive to the normality assumption,        can be applied, i.e.,        logit[Pr(Y ₁=1|X ₁ , . . . , X _(k))]=C(0)+Sum(j=1,k)C(j)X        _(j)  [8]        In general, the model generation platform matches the modeling        choice to the characteristics of the dataset and to the project        goal at hand.

The next stage 194 is to fit or train the model to the sample subset ofthe historical data using the predictive variables generated by a set ofvariable transformation to maximum univariate predictive capability anda set of dimension reduction filters to retain only the most predictivesubgroup of variables, including up to a level of interaction, forexample, tertiary interactions. At successive stages of convergence ofthe model parameters, the performance may be evaluated 198 and testedfor optimality 200. The evaluation is based on a set of criteriaincluding cumulative lift over the region of interest. If theperformance indicates the model is optimal then the model is persisted;otherwise, the sample dataset is enhanced 196 and the process iterated.The enhancement may take the form of higher degrees of interaction. Thecandidate models are not based on recursive partition/decision trees,which tend to overfit the data, but partition trees can suggestcontributory interactions that can be implemented as variable transformsusing the predictor variable editor.

Robust Model Assessment

A key to successful deployment of predictive models is the correctassessment of performance beyond the training sample. The modelgeneration platform employs validation and cross-validation and othermeasures to validate the model selection process that will yieldsuperior target classification performance in the marketplace.

As indicated in FIG. 12 the model generation platform can obtain theselection of validation datasets 210 by a variety of means such as apre-specified validation subset or the creation one or more from thedevelopment dataset by random sampling without replacement in a singlesession but with replacement across sessions. The operation of the modelgeneration platform is subject to condition-action rules, i.e., rules ofthe form:If <Condition>then <Action>[9]Warning messages are posted automatically, for example, if the number ofbuyers in the sample is less than 5% or less than 1000 in number, so asecond sample can be generated 218 or the model development processcontinued with the proviso that the model should be carefullyscrutinized for robustness.

For the sample dataset selected, cross validation is carried out by10-fold resampling with replacement in the decision/recursive partitiontree phase. After a candidate model has been determined in the modeldevelopment procedure, the model is applied to the holdout or validationsubset for comparison 212. Selected measures, for example, thesimilarity of the c-statistics (obtained from the ROC curves) and thecumulative and non-cumulative gain functions provide the basis forcandidate model validation 214. For particular variables in the model,the chi squared statistic can be used to rank order the significance ofthe key predictors. Variables that are significant above the 0.025 levelmay be sample dependent. To militate against that uncertainty, thevalidation set can be fitted independently and the common variablescompared. The congruence of the performances of the candidate model onthe development sample dataset and on the validation dataset based oncertain criteria validates not just the particular candidate modeltested for that dataset but the model development process for thatdataset 216. If the model generation process fails validation then theprocess is revised 220 using model persistence analysis. If the modelgeneration process is determined to be valid then the entire post-sampleselection model development process can now be applied to the entiredevelopment dataset, not just the sample dataset. The resulting finalmodel, tested for performance, which should be bounded by that of thesample dataset and by the validation dataset, will have superiorrobustness.

The validation (hold-out) gives an unbiased estimate of performance butwith considerable variability. Robust model assessment can be enhancedby bootstrap sampling for model parameter estimation, to expedite theprocess of linking multiple models. Automated cross-validation ofmultiple sample selections will identify the variables that are stronglypredictive across multiple samples.

A slightly biased but more accurate and complete approach is to performa complete k-fold sampling for the dataset of size N. However, fordiscrete valued variables, this is equivalent to a complete subsamplingof size T where T=N/k; a more computationally intense approach is toapply the candidate model to samples of size N, with 1 record chosenfrom the sample dataset and N-1 chosen from the validation (hold-out)dataset.

Model Persistence

For good targeting, the boundaries of validity of a single model'svariables should be recognized when different model variables need to bedeployed. This can be done by examining a decile-by-decile fit of themodel for significant loss of predictive power that would indicate thatan interaction with existing model variables has not been taken intoaccount.

The model generation platform automates a modeling process that leads toa candidate model for assessment. Part of that assessment can be anexamination of the scored and rank-ordered development dataset for anysignificant remaining differences between the population of targets(e.g., buyers) and the population of non-targets (e.g., non-buyers) forthe development predictor variables. On a decile-by-decile basis, themeans for the key predictor variables for the segmented groups can becompared graphically and by using the student t-test for significance.If there are significant differences, interactions with existing modelvariables may be called for as part of the revision of the modelgeneration process.

Construction of Model Ensembles with Data Weaving

Good predictive models often combine predictor variables from diversesources, such as demographics and transactions. The model generationplatform may combine variables from multiple sources and dealsimultaneously with multiple targets as indicated in FIG. 13. One typeof predictive model 232 may need to be combined 230 with at least oneother predictive model 234 of the same or different type. In suchsituations, it is important and appropriate to deal with responsepropensity in order to create cross-modal deciles. The data weavingfeature provides normalization for cross-product optimization. Automatedcross-product optimization will normalize, without analyst input,multiple models (e.g., product optimization) to compensate for differentresponse rates for each. This will expedite the process of linkingmultiple models.

The multi-stage models may also be applied, but are not limited to,concatenating models giving propensity of a selected action with riskmodels indicating likelihood of, for example, payment default, claimsrisk, or attrition. In another example, the multi-stage models can beapplied to concatenate predictions of propensity of a selected actionwith expected value models predicting, for example, usage level orretention duration. When the multi-stage models include predictions ofexpected value, aspects of those predicted propensities can be appliedin both the derivation of Expected Value or NPV 236 and the calculationsthat integrate economic data for expectation maximization 242.

The model generation platform using the metadata of the developmentaldataset can distinguish demographic variables from transactionvariables. Then using the model project variable editor, such variablescan be combined. Multiple targets can be accommodated by using as apredictor variable, a nominal variable with discrete values. Forexample, in a product optimization project, since the overall responserate is maintained in the project table of the database the response canbe normalized across products. The predicted product match for a givenindividual can then be made from the propensity for the product choices.However, normalization is required as shown here:

IfPr(Y _(i)=1|X _(ik) =x _(k))=exp(γ_(k0)+γ_(k1) ·x_(k))/[1+exp(γ_(k0)+γ_(k1) ·x _(k))]  [10]thenPr(Y _(i)=1|X _(ik) =x _(k))=[exp(γ_(k0)+γ_(k1) ·x_(k))/[1+exp(γ_(k0)+γ_(k1) ·x _(k))]s/F _(i)  [11]

-   -   where F_(i) is the frequency over sampling for product i.

For the appropriately computed propensity, economic data, such as thecost of sales, variable marketing cost, the cost of goods, purchaserusage level, and/or the expected retention by the purchaser, may beadded to derive the expected value or net present value of a potentialpurchase 236. Additional predictive models of one 238 or more types 240may be used to determine the predicted retention of a subscriptionservice, or the predicted length and magnitude of an interest bearingagreement, that will enable computation of profitability or the optimalstrategy to maximize profits using expectation maximization 242.

Customer insight The model generation platform provides a series ofactionable reports profiling customers' past and future behaviors. Inthe context of product marketing, these reports, derived from predictiveanalytics, yield a feature spectrum, for example, on customer segmentsfor generating creative marketing development, mass marketingpromotions, marketing planning, and product development.

As indicated in FIG. 14, the model generation platform includes afacility for computing customer insight based on ranked profiles of thekey predictor variables and comparisons of the averages for the keypredictor variables for buyers and non-buyers or other form of targetsegmentation. The insight process starts with the aggregation ofdatasets 252, such as transaction and communication data 250,demographic data 254, econometric data 256 and other data bearing oncustomer's past and future behavior. The aggregated dataset undergoestransformations of variables 258 that augment predictive power relativeto the selected outcome dependent variable. One method of ranking 260that sets aside the variable interactions is to using univariateregression analysis to compute the c-statistic from the ROC curve. Thisplots q over the range (0,1) forq=1−F(G ⁻¹(1−p))  [11]

-   -   where F and G are the cumulative frequency distributions of the        buyers and non-buyers. The rank order of the key predictor        variables derives from the magnitude of that AUC (area under ROC        curve). The semantic interpretation of that variable comes from        its description in the project data dictionary of FIG. 2. The        unweighted arithmetic means of the variables for buyers and        non-buyers are used for the graphical comparisons. The first        method may have the advantage of a simple interpretation of        predictor variables in terms of prospect attributes.

Another method uses the predictive modeling process to obtain a reducedset of relevant predictor variables 262 that generate a predictive model264 on that reduced set. The second approach has the advantage ofproviding estimates of future behavior and may be used to profilecoherent groups of prospects, not just past customers.

For the resulting sets of predictor variables the impact upon theprospect population must be computed 266 to obtain the most usefulranked set. For example, a particular variable may be highly predictiveof behavior but only expressed by a small subset of the prospectpopulation. In another example, economic data associated with theprofiled population can be utilized to compare the value, for exampleexpressed as net present value, of customer cohorts exhibiting affinityfor a designated product or service. One strategy is to rank thevariables in terms of size of impact, strength of predictability, andease of interpretation to obtain customer profile 268.

The Model Generation Platform Workflow Orchestration

The model generation platform includes a rule-driven analytic workbenchthat assists analysts in developing optimal predictive analytics modelsby automating data cleansing and merging, data transformation, modeldevelopment, model development process validation, model regenerationand refreshment, and generating and executing runtime models (which canalso run standalone outside the model generation platform). The projectframework for model construction incorporates all information needed forthe model but include datasets by reference only, records model designdecisions and comments for process improvement, and automates modeldevelopment, e.g., from templates and process flow.

The Advantages of the Project-Based Model-Generation Platform Includethe Following:

1) Rapid Model Development

In contrast to scripting tools often used for model development, themodel generation platform does not require the composition (ordebugging) of code but uses instead the context of a point-and-clickstructured graphical user interface. This lays the foundation for rapidmodel development and robust, reliable deployment.

The model generation platform automates the workflow underlying thedevelopment of a predictive model in a consistent, reproducible way. Toguide the process, interim real-time reports of the state of developmentare generated and retained as a permanent part of the model project. Toaccompany the scored list, an additional class of real-time reports isgenerated summarizing the interim reports and including the scoringfunction. Although the development dataset, scoring dataset, and scoringlist whose size warrants archiving are referenced by pathnames, thescoring function and all relevant information needed to refresh themodel on a going-forward basis are contained in the project folder.

2) Integrated Data Management

The model generation platform is designed for straight throughprocessing (STP) of customer data records (CDRs) from input, tomodeling, to scoring and the deployment of actionable lists. Forsensitive information, the model generation platform employs anencryption library so that, for example, the commercial standardTripleDES encryption is built into list scoring post-processing.

The model generation platform has access to a full suite of modules forimporting scrubbed data, transforming the variables into usable form,then generating customer insight and scoring lists

As shown in FIG. 15, the modules can be combined to generate astraight-through model generation platform pipeline for a particularproject workflow task using the project database to orchestrate theworkflow. Data preparation 270, which involves data cleansing, dataaggregation, and variable transformation, operates on historicaldatasets 272 as well as current datasets 273, with outcomes to becomputed 274 by one or more models 276 then applied, assemble a scoredlist 278, some or all of which may require encryption 280.

3) Scalability

Using a project-based model generation platform of the kind describedhere, as the number and complexity of modeling projects increase, thetime required for modeling, represented by workflow, data transformationand modeling does not increase beyond available resources. In typicalsituations it is not even a critical-path segment with respect toactivities of a business for which the modeling is being done, such as amarketing campaign.

4) Integrated Information Management

Efficiency is improved in the modeling process by enabling analysts toenter the modeling task bar process at any step to modularize themodeling process (e.g., dimension reduction used for quick evaluation ofa dataset).

Server pipelining may be enhanced to be able to handle a large number ofconcurrent active models and new products incorporating models.Scalability in the model generation platform server is built into thearchitectural design and configuration of the software and hardware. Oneexample of a model generation platform is as a multithreaded applicationwith component architecture that, for example, takes advantage ofmultiprocessor technology and runs under the cluster-capable AdvancedServer Windows 2000. Features that tune the software for scalableperformance include synchronous I/O support, concurrent scoring, servermultiprocessing, high availability, server clustering, and lights-outmanagement.

Model Development Process Flow

As shown in FIG. 16, predictive analytics is a business process withsequences of complex stages leading to a scoring model. The stages canbe represented as a sequence of actions performed on the data that guidethe subsequent processing. A variety of scenarios of use by an analystdepend on the task in which the analyst is engaged. For each of severalexample scenarios, the sequences of actions are discussed below. In FIG.16 the appropriate historical datasets are aggregated 290 and thenparsed 292 to obtain the model development dataset 296 and thecorresponding metadata 294 that is used by the model project duringmodel project set up 298 and throughout the modeling process. Thedevelopment dataset is first partitioned 300 to obtain a trainingsample. Then the training sample dataset is reviewed 302 at theindividual variable level 304 and transformed 306 or reviewed in termsof the set of predictor variables using partition binary trees andautomatically transformed 308 into variables suitable for dimensionreduction 310. For the reduced set of predictor variables a model methodis selected appropriate the data type and outcome 312 then applied tothe dataset 314. The validity of the model generation process isevaluated 316. If the process does not validate the dataset and modeltype are revised. If the convergent process validate then the finalmodel selection is applied 318. The model process results are displayedin report form 320 or lead to actionable model ensembles 322. Theresults of prescoring validation 324 can be reviewed as a report andpersisted before making the commit for a score list 326.

Model Development Guide

Corresponding to the workflow of FIG. 16 is an iconized representationviewable by the user in stages that serves as the overall interactivemodel development process guide as shown in FIG. 17. The diagram showsthe development path starting from dataset selection 340 and theaccompanying metadata dictionary selection 342 as part of the modelproject initial entry 344 corresponding to 298 of FIG. 16. The nextstage, corresponding to 300 of FIG. 16, is the partitioning 346 ofdataset into a training sample 350 for the model and one or morehold-out samples 348. The subsequent stage, corresponding to thepredictor variables management 302 of FIG. 16 and its auxiliarysubstages 304, 306 and 308, is the variable selector 352. Directlycorresponding to the dimension reduction 310 of FIG. 16 is 354 of FIG.17. The choice point 356 indicates the options available at thedimension reduction stage 310 of FIG. 16 to apply more stringentfiltering, to revise the pool of predictor variables 302 or to advanceto model selection 312. The model selection stage 358 in turn providesoptions 360 corresponding selecting a different model from the libraryof appropriate models 312, to revision of the pool of predictorvariables 302, if the model convergence 314 did not attain projectgoals, to validation of the model generation process 316. Subsequently,the final selection stage 362 includes the validation of the modelgeneration process 316 and final model selection 318. Following thefinal select 362 the choice point represents the options of generatingreports 368 corresponding to step 320 of FIG. 16, or generating morecomplex model with model ensembler 322.

In addition to the rule-guided canonical path of FIG. 17, the analystcan make use of the re-entrant properties of the interactive processguide. The candidate model can be revised or a new model projectinitiated by clicking on the underlined hyperlink labels. The analystcan double-click on any of the underlined hyperlink labels.Double-clicking on the dataset hyperlink enables changes in theselection of the dataset file; similarly, for the dictionary hyperlink.Double-clicking on the model entry hyperlink enables viewing or editingof the model project description. Double-clicking on the highlightedpartition hyperlink allows the analyst to progress to the next step inthe model development process. The model generation platform embeds aknowledge/experience base in the choice points and paths.

Model Generation Platform Launch: Analyst Creates a New Model

In this scenario, the analyst launches the model generation platformapplication. The immediate result is the launch of a new model projectresulting in the display of the model entry form dialog as shown in FIG.18A, which, for convenience, is an example for a binary response model.

Model Project Initialization

The model project entry form of FIG. 18A is used to initiate the modelproject by collecting basic information to track and document themodeling development process. The model project entry form first obtainsbackground information about the model project such as the designatedname, the type of model to be developed, and the objective to focusupon.

The model project is based upon an associated dataset, which has anassociated dataset dictionary. The user can browse to the file path forthe development dataset and to the development dataset by clicking onthe browse datasource button 370 and the browse dictionary button 372.Open file dialogs are used to choose the appropriate file paths.Selection of the data dictionary database (e.g., DataDictionary1.mdb)displays the dictionary table on the entry form as shown in FIG. 18B.From among the variables in the particular dataset, one variable shouldbe selected as the outcome variable [dependent variable], e.g.,Current_purchase, which can be predicted from the remaining availablepredictor variables. The user may select that variable by clicking onthe appropriate row, which causes the variable to appear in thedependent variable dropdown list 374. The selected variable is nowidentified as the dependent variable (and is excluded from thepredictors, i.e., Status is XP).

After completing the selections for the candidate model properties andmodel constraints which serve as descriptive guides or embedded filtersparameters for selecting a final model, the user clicks on the submitbutton 376 to merge the dataset dictionary and dataset into the internalrepresentation and compute basic statistics.

After the analyst clicks on the submit button 376, the model generationplatform sets up the project structures (including the project table)and performs consistency checks, for example, matching the datadictionary and the primary dataset. After the metadata has been appliedto the development data set, the iconized development guide, shows thecompleted steps and indicates the next step, partitioning. Completedsteps are linked by lines of one color index (green) but currentincomplete steps are linked by a second color index (red). The guidestarts with the earliest steps that have been performed and ends on theright with the next step to be performed, rather than showing allpossible future steps.

Splitting Sample Dataset from Validation Datasets

The next stage involves creating sample datasets by random sampling. Themodel generation platform then creates views of the original datasetthat constitute the sample and validation dataset(s) and represents themin the iconized workflow display within the two boxes for the hold-outand sample datasets. The model generation process automatically advancesto the next stage of model processes, that of data exploration,understanding and transformation, where the tools of the platformsynergistically interact with the experience of the analyst and providevariable transform automation.

Dataset Exploration, Understanding and Transformation

The model development process subsequently examines overall significanceof the variables and their potential interactions. Even if the datasetspresented to the model generation platform have been aggregated andsplined there may be nonlinearities and interactions that can contributesignificantly to the model. The predictor variables palette shown inFIG. 19 provides a way of examining such interactions:

Predictor Variable Review

Next step 4 is variable selection and editing. The analyst is presentedwith the predictor variables palette of FIG. 19, which shows thepredictor variables that survived the filtering for non-contributoryvariables (in the column headed “predictor variables”), together withthe non-contributory variables, which either are in original form(headed “excluded variables”) or generated by a construction proceduresuch as a decision tree split, OLS product, or cross product (column“excluded constructed variables”).

This palette allows the analyst to examine individual variables,construct additional candidate variables, and classify each variableinto one of the three columns. By selecting a variable (or by clickingon the select all button), the analyst can use the arrow buttons toredistribute the variables. Only the predictor variables in the centercolumn are carried forward in the subsequent steps of the modeldevelopment process.

The predictor variable palette shows the status of the variables in thesample dataset (FIG. 19). Except for initially excluded variables asindicated in the dataset dictionary, such as CustomerID, all the primaryvariables are designated as predictors (“P”). When a dependent variableis selected or a predictor from the set of primary variables, its statuschanges to excluded primary variable (“XP”). Such variables aredisplayed in the right hand list. Clicking on one of the variables inthe list presents the basic features of that variable in the variableeditor.

In FIG. 19, clicking on the reduce dimension button presents the analystwith the dimension reduction dialog. If dimension reduction is notrequired, the user clicks on the reconsider model button to display themodel selection palette of FIG. 23.

After creating one or more new transformed variable, the user clicks onthe revisit reduction button to return to the dimension reduction stageto refilter the set of predictor variables but the result may beunchanged.

Alternately, the analyst can reprocess the variables by clicking on therevisit (dimension) reduction button to reduce the dimensionality of themodel dataset using the newly created predictor variable in the pool ofcandidate variables. Either the same or a different reduction method canbe used or further reduction can be omitted to progress to the modelselection process of determining the best predictor variables and themodel regression equation that provides the response prediction.

The predictor variables palette dialog of FIG. 19 presents the analystwith three columns representing the three functional states of avariable for the modeling process. The center list contains thosevariables that are the potential predictor variables of the model. Mostof the dataset variables will initially be in that potential predictorstate represented by P in the Status column of the Data Dictionary. Theleft-hand list will display transformed variables, i.e., thoseconstructed by transforms of other variables; however, specificallythose transform variables that have been temporarily excluded from theset of predictor modeling variables. The right-hand list will display asecond set of variables excluded from the set of predictor variables,such as the dependent or target variable. When the analyst clicks on avariable in one of the lists the information from the Data Dictionaryassociated with that variable is displayed below along with a hyperlinkto graphical information on that variable.

The predictor variable palette shows the status of the variables in thesample dataset (FIG. 19). Except for initially excluded variables asindicated in the dataset dictionary, such as CustomerID, all the primaryvariables are designated as predictors (“P”). When a dependent variableis selected or a predictor from the set of primary variables, its statuschanges to excluded Primary variable (“XP”). Such variables aredisplayed in the right hand list. Clicking on one of the variables inthe list presents the basic features of that variable in the variableeditor.

Variable Visualization

In addition to the information from the data dictionary, the selectedvariable is graphed in terms of the response frequency of the targetdependent variable. By clicking on the hyperlink the graph is displayedas shown in FIG. 20B:

Such variable visualization provides a basis for transforming variablesto predict more robust target performance. This is particularlyeffective for nonlinear continuous variables, for multi-valued nominalvariables whose effectiveness can be improved by splining, and forvariables with significant interactions with other predictor variables,such as those revealed by the partition tree plots.

Variable Transformation

For a selected variable, the major variable transforms (logarithm,square, square root, interaction) can be applied. To obtain aninteraction transform, the user clicks on the edit button then on theinteraction (X) button as shown in FIG. 21A. Then, in the predictorvariables list, the user clicks on the prior_contact_before_purchaselvariable to generate the transform variable shown in FIG. 21D. Theentries are automatically generated but the variable description can beedited manually. After clicking on the save button and new variablePrior_Purchase_I appears on the predictor variable list which whenselected shows the new set of features as shown in FIG. 21E:

After creating one or more new transformed variable, the user clicks onthe revisit reduction button to return to the dimension reduction stageto refilter the set of predictor variables but the result may beunchanged as shown in FIG. 22. Action point 3 is variabletransformation. By double-clicking on a variable in the variablepalette, all the information about that variable can be displayed,including its definition as shown in FIG. 21A. The analyst can modifythat definition to create a new variable by using a functionaltransform, e.g., logarithm or square root, window mappings to excludeoutliers, e.g., x<13.0, or splines. Splining is a special case ofvariable transformation and has an auxiliary window for graphicallydetermining the knots.

Distribution analysis is available for the variable in terms ofdescriptive statistics and graphs and relative to the target variable.Moreover, automatic partitioning of a single variable into morepredictive components can be carried out or the interactions amongsignificant variables in the full predictive set can be revealed. Forthe subset of predictor variables under consideration a descriptiveanalysis can be obtained by selecting the first distribution analysischeckbox; differential effects can be examined by selecting on thesecond distribution analysis checkbox. The user may select theinteraction tree checkbox to examine potential variable interactions.

When the displays have been computed, hot links to the distributionanalyses and decision trees in HTML format are shown on the palette asillustrated in FIG. 20A.

Clicking on an exploratory statistics hotlink launches a browser todisplay the linked HTML as shown in FIG. 20B for the sample variableinteraction tree.

The partition analysis shows a graphical representation of the 50,000records (non-buyers at the top and buyers at the bottom) using the mostsignificant variables as determined by a log likelihood measure whosebifurcation will split the sample into a binary decision tree.

Since the partition beyond a certain depth often overfits the sampledataset, a restriction is place on the procedure so that the small halfof a split must contain at least 10% of the parent set.

To test for robust generalization, a k-fold cross-validation isperformed using the log likelihood (G²) statistic to indicate overallfit. There is a trade-off between number of subsamples and randomvariability (noise) but typically between 5 and 10 is satisfactory. Inthe model generation platform, a 10-fold cross validation is performedand by that test, the partition tree is robust.

After the variable have been examined or edited, sample frequency and/orinteraction displays have been reviewed, the user clicks on the proceedbutton to advance to the dimension reduction stage. As a backgroundtask, the model generation platform can impute missing data for suitablecategorical and continuous variables, when requested, apply BayesianRenormalization and then and use the G²-values to optimally partitionhighly multi-valued categorical variables.

Dimension Reduction Stage

Next step 3 is dimension reduction. The challenge posed to the analystby the complex, multidimensional dataset is to isolate the significantpredictor variables for use in a model. Although the dataset has beencleansed, information is not distributed evenly across the variables(dimensions). The process of exploratory data analysis seeks tounderstand the distribution of buyers and non-buyers across thedimensions, to facilitate the transform of variables since thepropensity mapping may be simplified with in a linearized space. Then tomake the model generation more efficient, dimensions irrelevant to thepropensity mapping can be eliminated or weakly contributory dimensionscombined. So dimension reduction acts to reduce the number of dimensionsby excluding variables that are not relevant.

There are three general approaches to reducing the dimensions of thespace in which to develop a predictive model, which are made availableto the analyst in the dialog box shown in FIG. 22. First, theobservational data may be lacking for certain covariates rendering theminefficient. When missing data are not ignorable and cannot be reliablyimputed then the corresponding sparsely populated covariates below somecutoff point may be discarded without loss in generality for afirst-pass model. Second, ordinary least squares regression (OLS)applied to the Bernoulli distribution of the dependent variable forlinear, quadratic forms of individual covariates can be selected usingthe probability of the t-ratio greater than some relaxed criterion, suchas 0.05. Discarding the variables that fall below the criterion willreduce the dimensionality of the model space and in this lowerdimensional space of significant variables; the interaction crossproducts can be similarly sorted and culled. However, the dependentvariable may be very non-linearly related to one or more covariates somay be prematurely set aside, if the cutoff criterion is set too high orthe relationship is too nonlinear. Third, the decision tree of the dataexploratory stage will have suggested any complex interactions as suchmachine learning classifiers can deal with complex regions withincovariates but may over fit the observations, if the trees are expandedtoo deeply. Therefore, restricting the nodes to those of a certain sizeabove some cutoff point will generate potentially significanttransformed covariates.

When the analyst selects the reduce dimension button after adjusting thedensity cut-off indicator level, the variables that fall below thecut-off line are excluded from further modeling consideration. (Suchvariables receive the status ‘XP’ but later in the process withadditional missing data imputation; the analyst can re-include thevariable based on other considerations.) A count of the number ofremaining variables is displayed for confirmation.

Through a set of filtering procedures indicated below the purpose ofthis stage is to exclude a sufficient number of the non-significantvariables that the modeling can proceed efficiently with procedures,such as stepwise OLS (Ordinary Least Squares) regression. In addition tothe first filter that eliminates sparsely represented variables thereare other variables that may include: a second that uses stepwiseforward OLS with the t-ratio as the fitting statistic (Prob |t|<cut-offcriterion, e.g., 0.05); a third, that uses stepwise forward OLS with allbinary interaction terms (Xj*Xk) filtered with the same technique; afourth that generates by forward stepwise OLS all combination of theaugmented sets of variables, X*, summed with the set, Z, initiallyrejected variables, e.g., Xh or (Xj*Xk) filtered with the sametechnique; a fifth one that generates all interactions of X*+Z with Z,using stepwise forward OLS filtered with the same technique with all asused as a cut-point or a ranking measure; with all terms summed with theinitially rejected variables filtered with the same technique. After theparticular filter regimen has been selected in FIG. 22, the user clickson the reduce button or hits the return key. Upon completion, a count ofthe number of remaining significant predictors is shown as in FIG. 22.To review the status of the predictor variables, the user clicks thereview predictors button.

Model Generation

Next step 4 is the prediction method. When the analyst double-clicks onthe model select icon, a selection of several model-building approachesis displayed as shown in FIG. 23A. The analyst progresses by choosingthe prediction method (and the criteria to exclude non-contributoryvariables), e.g., stepwise logistic regression. The modeling results aredisplayed using hyperlinks and if the criteria for a satisfactoryCandidate model are met, e.g., c>0.7 then a proceed to validation buttonis clicked. If the Candidate model is accepted, all variables except forthe predictor variables used in the model are moved to one or other ofthe excluded variable columns. If a Candidate model is rejected, thedata exploration and variable transformation action points are stillavailable for modifying variables to improve the model.

As an example, to select a method, the user clicks on the stepwiselogistic regression method radio button, then, selects the fittingcriterion by selecting the maximum likelihood method. For stepwisefitting there are three approaches: forward, backward, and mixed. Forexample, the analyst can elect the backward approach, in which all thevariables are entered but then eliminated if not significant, byclicking on the backward radio button. Then set the entry significancelevel combo box and the retention significance level combo box levels tosay 0.25 and 0.025. For the backward approach the critical level is theretention level, which should be set at 0.025 or below since at the0.025 level, a spurious variable with no actual predictive power will beaccepted with a 2.5% probability. After completing the parametersettings for generating a candidate model, the user clicks on thegenerate model button (FIG. 23A).

The model selection method for the dataset example uses stepwiselogistic regression to obtain an optimal set of coefficients for thevariables that are significant at the 0.025 level or below. When theprocess is complete, links appear to the results. The user clicks on thefirst hyperlink to have the browser display the model candidatestatistical results, specifically, the model parameter estimates and theROC Curve, which gives the concordance statistics as shown in FIG. 23B.

Click on the second hyperlink, Model_Candidate_Lift_Chart, to displaythe non-cumulative gains chart for the dataset sample.

Model Gains for Sample

The model candidate lift chart shows a significant lift in the topdeciles and a smoothly declining lift overall. The user clicks on thethird hyperlink, Model_Candidate_Persistence_Chart, to display the Keyvariables for the dataset sample, for example, as shown in FIG. 23C.

The persistence chart for this variable indicates that there is nosignificant interaction with other variables in the model that has notbeen accounted for. Because the model shows promise, to test further theuser clicks on the advance candidate button.

Alternately, if the model candidate did not meet the necessary criteria,clicking on the review predictors button will display the predictorvariable palette (FIG. 19) where new transformed variables can beconstructed to attempt to develop an improved model candidate.

Model Generation Process Validation

FIG. 24 shows the iconized guide view of the choice points for returningto review the predictor variables if the model is not satisfactory. Nextstep 5 is the validation of the model generation process. If the(working) model meets the criterion for a satisfactory model then theanalyst can proceed to determine if the model is sufficiently robust, sothat when extended beyond the Sample dataset used, the model retains itpredictive power. In the modeling process validation stage the analystis presented with a means to determine the validity of the process usedto generate the model.

The analyst is presented with a final model candidate selection dialogas shown in FIG. 25A. Similar to the sample selection process, theanalyst picks a validation dataset The complement of the sample set,hold-out set gives the most statistically unbiased single estimate ofthe robustness of the candidate model; however, using cross-validationtechniques, sets selected by resampling, although optimistically biased,give a more accurate estimate of the variability of robustness. Inaddition there is a default set of tests for robustness and validationthat can be augmented.

Validating the model generation process involves a comparison of theproposed model candidate features to the initial entry information onthe development dataset and the targeted values of the key featurestogether with a validation of the candidate using the hold-outvalidation dataset as shown in FIG. 25A.

Clicking on the validate button executes the mandatory and optionaltests as shown in FIGS. 25A and 25B. The user reviews the final modelcandidate selection checkboxes, and then clicks on the compare button toinitiate the comparison of the model candidate using the sample datasetand the validation dataset as shown in FIG. 25B.

After the comparison process completes with all three checkboxesselected, three links to the results are displayed as shown in FIG. 25A.

Clicking on the comparative model statistics button compares theconcordance (area under ROC Curve) for both the sample dataset and thevalidation dataset as shown in FIGS. 25B and 25C.

Comparison of Model for Sample and Validation Datasets

The concordance statistic (c) [area under curve] of the validationdataset is less than that of the sample dataset but still exceeds thelower bound of 0.70. Clicking on the cumulative lift chart buttondisplays the gains relative to those of an average random sample. Boththe validation and sample cumulative lift are similar. Clicking on thenon-cumulative lift button show similar performance

Both the sample and validation lifts are similar and drop monotonically.After reviewing the results generated by clicking on the compare button,there are decision options to consider as shown in FIG. 25A.

If the candidate model does not generalize sufficiently well or fails tomeet other criteria, the model method can be reconsidered by clicking onthe reconsider model button, or the set of predictor variables can bereconsidered and edited by clicking on the review predictors button.

On the other hand, if, as in the illustrative example, the candidatemodel is acceptable as the final candidate model, the analyst clicks onthe accept button to complete the model development process byregenerating the model using the full development set and thengenerating a report.

Final Model Candidate

The validation of the Candidate Model is the validation of the modeldevelopment process for the development dataset under consideration. TheCandidate Model was developed using just a subset of the fulldevelopment dataset. Validation of the model development process forthat development dataset enables the analyst to apply the same processto the full dataset with a resulting increase in accuracy and predictivepower. To complete the model development process the full developmentdataset is subjected to the same process, resulting in a final candidatemodel and that model is applied to the sample dataset and to thevalidation dataset with the final candidate model equation and theresults displayed in a report.

Report Generation

The model project report summarizes the description and goals enteredwhen the project was initiated together with the results of statisticaland graphical tests performed in the course of developing a model andthe model equation to be used for scoring as shown in FIG. 26. Now theaccepted final candidate model can be deployed to score a prospect listor develop insights about the customer base.

List Scoring

Prior to processing a prospect list, there are certain tests to beperformed to make certain that the model is appropriately matched to theincoming list. There may be recodings of the variables in the incominglist, e.g., household income in dollars rather than thousands ofdollars, but such modifications are easily accommodated. Moresubstantially, the model may be based on aged data so the scores areless than optimally predictive. Depending on the circumstances, one ormore tests shown above may need to be carried out. Clicking on thecompare button launches the computation of the tests selected resultingin up to four hyperlinks.

Activating the compare file variable statistics hyperlink displays thefollowing. In this side-by-side comparison, the two datasets havecomparable patterns of average variable values and standard deviationsfor the key predictor variables indicating that scoring should berobust. As the charts demonstrate the variables in the development fileand in the target scoring file display similar responses across deciles,which indicates that the development dataset is representative. As athird comparison, the distributions of the score computed by the modelcan be viewed in terms of comparative scores distributions

The score distributions are equivalent so the final model can becommitted to scoring the input list and producing the scored list. Thescores can be assigned to predefined deciles, either from norms or fromthe development dataset.

Customer Insight

FIGS. 27A through 27D are an exemplary graphical user representation ofa model project insight interactive dialog showing hyperlinks to atarget profile and corresponding chart of key factors; exemplarygraphical user representation of a model project insight chart showingtarget and non-target contributions for two key factors.

Good models can be used to gain insight into the customer base toefficiently target products to a given market. By clicking on theinsight button, the analyst launches the customer insight module shownin FIG. 27A.

Such an analysis is most useful when the predictor variables are highlypopulated. For the goal of constructing a predictive model, a cut-off,by default set to 5%, was imposed to filter out sparsely populatedvariables. However, promotional marketing imposes a higher cut-off,typically 50% or more. After setting the lowest acceptable match ratedropdown list, selecting on the analyses needed, then clicking on theget insight button, the model generation platform generates two types ofresults as shown in FIG. 27B:

Clicking on the display profile key factors hyperlink, displays in rankorder the key predictor variables as shown in FIG. 27C. Such keyvariables can be particularly enlightening when examined across productsor in detail as shown in FIG. 27D.

The customer insight module benefits from the construction of anappropriate model but can operate as well on a subset of the variables,for example, those representing demographics alone. Typically,transaction variables are stronger predictors but for some campaign,e.g., a cross-sell, demographics may be the most appropriate to gaininsight into clone populations.

Analyst Creates a New Model from a Template

In this scenario, next step 1 is to import a new dataset. The analystopens an existing model project, then selects import dataset and choosesa dataset requiring the same type of model, e.g., a later phase of amarketing campaign, and uses the same data dictionary as shown earlier.

Next step 2 is a repopulation of the datasets. The model generationplatform repopulates the sample and validation datasets, updates themodel project entries including a new (default) model project name, andprogresses through the modeling process to the modeling results displayof FIG. 26.

Analyst Completes Model from Previously Saved Model

In this scenario, next step 1 is the prediction method. The analystopens an existing model project that has been previously saved as a workin progress. The model generation platform repopulates the sample andvalidation dataset partitions, the subsets of predictor and excludedvariables including any transformed variables as shown in FIG. 19. Theanalyst can then select the prediction method to use for modelcompletion.

Analyst Revises Completed Model

In this scenario, next step 1 is the prediction method. The analystopens an existing model that had been finalized which then forces a saveunder a different name. The goal is to improve on the model or import anew dataset using the final model as template. The model generationplatform repopulates the sample and validation dataset partitions, thesubsets of predictor and excluded variables including any transformedvariables, executes the previously selected prediction method and thefinal tests and displays the results (FIG. 25). The analyst can thenreject the final model and re-analyze the model starting with predictionmethod/import dataset.

Next step 2 is combined models. For particular distributions of targetsin the dataset, a single model may not be optimal. A combined model,e.g., one optimal for the low deciles and a second model for the topdeciles is the solution to such a challenging situation.

The model generation platform is designed to develop a finished model,document the development, and provide statistical and graphical reportsof that development. The goal is to automate the majority of steps instatistical manipulations letting the analyst decide what criteria toapply, e.g., cut-off level to reduce the dimensionality of a dataset,and then judge the sufficiency of the results to determine progressiontoward validation of the model development process and the production ofa final candidate model. A major design target is to make completion ofthe final model as fast and efficient as practical.

Accordingly, in view of the above; the model generation platform 10illustrated in FIG. 1 and described herein is readily implemented usingdata and information available to software engineers that are skilled inthe art, and the disclosure herein is adequate and provides sufficientinformation to permit one skilled in the art to practice the presentinvention.

Other implementations are also within the scope of the following claims.For example, variations in the identity and the sequence of the steps,choice points, and action points described above are possible. Otheruser interface approaches and elements may be used. The mathematicalunderpinnings of the system may be different than those described in theexamples. The system can be applied to a variety of modeling projectsother than customer behavior in a consumer or business market. In thecontext of customer behavior, as it applies to consumer and businessmarkets, the platform can be applied to all stages of the customerlifecycle from customer acquisition, through customer development, tocustomer retention. The role of these applications may be to predict,monitor, or compare behavior, and the intent of the applications may beto elicit, monitor, reinforce or change a behavior. The platform mayalso be applied to assess a population of individuals, for example, toforecast trends or the magnitude of a future outcome stemming from thepopulation. In another example, the platform may be applied to calculatea population's economic value, credit standing, or orientation toward adesignated area of interest.

The distribution of the functions and components need not be as shown,but can instead be distributed over any number of computers or networks.Additionally, although we use the terms client and server, any givenprogram may be capable of acting as either a client or server; our useof these terms may refer only to the role being performed by the programfor a particular connection, rather than to the program's capabilitiesin general.

Seven other applications being filed on the same day as this one andwhich share a common detailed description, are incorporated byreference.

1. A machine-based method comprising in connection with a project inwhich a user generates a predictive model based on historical data abouta system being modeled: selecting variables having at least apredetermined level of significance from a pool of potential predictorvariables associated with the data, to form a population of predictorvariables, extending the population to include non-linear interactionsof variables, extending the population to include linear and non-linearextensions with remaining previously excluded variables, generating apossible model of the extended population of variables using a subsampleof the data, determining whether the model generalizes to the data otherthan the subsample, if so, applying the possible model to all of thedata to generate a final model, and cross-validating the final modelusing random portions of the data.
 2. The method of claim 1 alsoincluding displaying information to the user of the size of the pool ofpredictor variables.
 3. The method of claim 1 also including enabling auser to point and click to reduce or extend the size of the pool ofpredictor variables, retaining only the most significant predictorvariables.
 4. The method of claim 1 in which the user is enabled toinvoke an automatic process to select a class of models most suitable tothe pool of predictor variables for the associated dataset.
 5. Themethod of claim 1 in which the user is enabled to point and click toadjust the model selection criterion to retain only the most potentvariables for the target goal.
 6. The method of claim 1 in which theuser is enabled to point and click to cause display of information aboutthe model performance.
 7. The method of claim 6 in which the informationabout the model performance includes at least one of: a statisticalreport card with a link to the statistical report chart, a lift chartwith a link to the lift chart, a response comparison chart for eachdecile for each predictor variable in the model, and a link to theresponse comparison chart.
 8. The method of claim 7 in which invocationof the link to the statistical report card causes display of thestatistics of model performance.
 9. The method of claim 7 in whichinvocation of the link to the lift chart causes display of anon-cumulative lift chart.
 10. The method of claim 7 in which invocationof the link to the response comparison chart causes display of aresponse chart for each predictor variable in the model for each segmentof interest.
 11. The method of claim 1 in which a user is enabled tochoose interactively at least one performance criterion change ortransformation or interaction of variables to improve the fit of themodel.
 12. The method of claim 1 also including a enabling adetermination whether the model generalizes to the data other than thesubsample, and, if so, applying the possible model to all of the data togenerate a final model, and cross-validating the final model usingrandom portions of the data.
 13. The method of claim 12 in which theuser is enabled to select at least one validation dataset and invoke amodel process validation method.
 14. The method of claim 12 in which theuser is enabled to point and click to cause display of information aboutthe model process validation.
 15. The method of claim 12 in which theinformation about the model process validation includes at least one of:a statistical report card with a link to the statistical report chart, acumulative lift chart with a link to the cumulative lift chart, anon-cumulative lift chart with a link to the non-cumulative lift chart.16. The method of claim 1 in which the user is enabled to select atleast one machine automated model development process applied to theentire dataset for a validated model process.
 17. The method of claim 1in which the user is enabled to point and click to cause display ofinformation about the performance of the validated model process appliedto the entire set of historical data.
 18. The method of claim 17 inwhich the information about the model performance for two independentdata subsets includes at least one of: a statistical report card with alink to the statistical report chart, a cumulative lift chart with alink to the cumulative lift chart, a non-cumulative lift chart with alink to the non-cumulative lift chart.
 19. The method of claim 18 inwhich the invocation of the link to the statistical report card causesdisplay of the statistics of model process validation.
 20. The method ofclaim 18 in which the invocation of the link to the cumulative liftchart causes display of a cumulative lift chart.
 21. The method of claim18 in which the invocation of the link to the cumulative lift chartcauses display of a non-cumulative lift chart.
 22. The method of claim18 in which the final model and the model process validation results arestored persistently.
 23. The method of claim 1 also including enablingthe user to observe the number of predictor variables available for themodel.
 24. The method of claim 1 in which a model method from a libraryof model methods most suitable to modeling the historical data set isautomatically selected.
 25. The method of claim 1 also includingenabling the user to observe the performance of the model by means oflinks to a plurality of statistical and graphical reports.
 26. Themethod of claim 1 also enabling the user to select a means of validatingthe model development process.
 27. The method of claim 1 also enablingthe user to observe the performance of the model for the training andvalidation subsets of the historical dataset.
 28. The method of claim 1also enabling the user to invoke at least one validated modeldevelopment process to produce a final model enabling the use to observethe performance of the final model on at least two independent subsets.29. The method of claim 1 enabling the persisting of the final model andintermediate results to the project database.
 30. The method of claim 1enabling the final model to be applied to scoring at least onenon-historical dataset wherein the propensity computed by the model isindexed by the score.
 31. A machine-based method comprising inconnection with a project in which a user generates a predictive modelbased on historical data about a system being modeled, displaying to auser a lift chart, monotonicity, and concordance scores associated witheach step in a step-wise model fitting process.
 32. The method of claim31 also including enabling the user to observe changes in the fit of themodel as variables associated with the data are added or removed from apredictor set of the variables.
 33. The method of claim 31 alsoincluding enabling the user to terminate the fitting of the model whenthe fitting process reaches an optimal point.
 34. A machine-based methodcomprising receiving from separate sources, sets of variablesrepresenting historical data about a system being modeled, and enablinga user of a model generation tool to combine at least two of thevariables from the sets of variables.
 35. The method of claim 34 inwhich enabling the user to combine the variables includes providing auser interface that enables the user to identify the variables to becombined.
 36. The method of claim 34 in which the system comprisesbehavior of prospective or current customers with respect to products orservices of a company and the method also includes adjusting outcomevariables to normalize response rates across products or services withdifferent response rates.
 37. The method of claim 34 in which the systemcomprises behavior of current customers with respect to retention of acurrent service or product of a vendor and the method also includesadjusting variables to normalize response rates across products orservices with different response rates, using the computed propensitiesas indices of the scores.
 38. The method of claim 31 also comprisingdetermining a course of action to yield the most positive NPV outcome.39. The method of claim 38 in which the determining includes selectionof a mix of channel and product combinations.
 40. The method of claim 38in which the determining includes predicting retention in combinationwith response rate to predict NPV.